Imaging systems and methods for tracking objects

ABSTRACT

A first imager has a relatively high resolution and a relatively narrow first field-of-view. Information about objects in an environment is detected or captured, and used to steer the first field-of-view of the first imager. The sensor(s) may take the form of a second imager with a relatively lower resolution and relatively wider second field-of-view. Alternatively, other types of sensors, for instance presence/absence sensors may be employed. The first field-of-view may be directed toward an object that satisfies one or more conditions, for instance matching a particular SKU. The first field-of-view may track a moving object, for instance via a tracking mirror and actuator. This approach may be employed in retail locations, for example in grocery or convenience stores, for instance to reduce various forms of theft or in industrial environments.

BACKGROUND Technical Field

The present disclosure relates to imager based systems, for exampleimage-based systems that automatically detect, monitor and track objectsin an environment, for instance in a retail store.

Description of the Related Art

Many applications would benefit by automated detection, monitoring ortracking of objects in the respective environment. For example, retail,warehouse, baggage sorting, parcel sorting, or gaming casinoenvironments may benefit by automated detection, monitoring or trackingof objects using image-based techniques.

In many applications or environments, objects to be monitored or trackedare moving or in transit. Camera systems designed to capture images ofmoving objects typically employ a global shutter imager with shortexposure time, and often employing pulsed illumination.

For example, a retail environment (e.g., grocery or convenience store)may employ ceiling mounted cameras used for detection of items inshopping carts. Such cameras must be capable of capturing relativelyhigh-resolution images in order to discern the specific objects oritems. In particular, machine vision algorithms require a certain numberof pixels in order to properly identify objects or items. Thus, when alarge area is to be covered, a camera or imager mounted in a ceilingwould have to have a very high resolution. As an estimate, in order torecognize items as small as 5 inches by 5 inches in size (e.g., a box ofTide® laundry detergent), an image would require approximately 200pixels by 200 pixels. At approximately 40 pixels per inch, a 5 Megapixelimager could only cover an area of about 5 feet by 4 feet while stillproviding sufficient resolution to identify objects or items. Forexample, if the application is monitoring or tracking objects or itemsat a checkout station or counter of, for instance a grocery store (e.g.,on a conveyor belt at the checkout station or counter, and/or in theshopping cart), this area of coverage is not sufficient. In particular,a position of a shopping cart at a checkout station or counter can varysubstantially from one checkout transaction to another, or even during asingle checkout transaction, making selection of an appropriate 5 footby 4 foot area for imaging virtually impossible.

Further, sharp image capture is needed for successfully reading linearor one-dimensional machine-readable symbols (e.g., barcode symbols)and/or two-dimensional machine-readable symbols (e.g., area or matrixcode symbols). For moving objects, sharp image capture typicallyrequires very fast shutter speeds, which can be impossible using ambientlight, leading to the need for expensive and large active illuminationsystems. Active illumination can interfere with other opto-electroniccomponents of a system or device, and may present an annoyance to humanoperators.

BRIEF SUMMARY

Global shutter imagers typically have lager pixel size, and are moreexpensive and noisier than rolling shutter imagers.

Applicants have realized that a smaller, less expensive camera systemthat requires less light to operate can be implemented, for example,if: 1) an area to be imaged is kept small, and/or 2) relative objectmotion can be minimized. Such may advantageously allow for lower cost,smaller size, faster tracking, and lower levels of required illuminationthan existing global shutter imager-based approaches.

In at least one implementation, a field-of-view of an imager or camera,for example a rolling shutter camera, is directed or caused to track oneor more objects via a steering mirror, for instance a fast steeringmirror. A tracking subsystem may include a tracking imager or camera, orsome other sensor(s) or transducer(s) that capture data (e.g.,3-dimensional data) that characterizes objects in an environment, forinstance a retail environment. In other implementations, images capturedby a first imager may be used to determine characteristics of objects,which characteristics are used to cause a respective field-of-view ofthe first imager to track one or more objects. Characteristics may, forexample, include appearance, presence, location, position, speed, and/ordirection of travel of the object. Characteristics may also, forexample, include physical characteristics of the object and/orpackaging, which characteristics allow classifying the object as acertain type of object (e.g., stock keeping unit or SKU, restricted saletype item). The steering mirror directs the field-of-view of arelatively higher resolution imager with a relatively narrowfield-of-view to track an object, for example an object spotted in awider field-of-view of another imager, and/or an object that is inmotion. For moving objects, relative motion between the object and thefield-of-view of the higher resolution imager is reduced or eliminated,allowing use of a rolling shutter imager or camera.

Image sensors of rolling shutter imagers typically have smaller pixelsthan image sensors of global shutter imagers. Rolling shutter imagerstypically have lower noise due to the ability to cancel kTC noise, whichmay otherwise be a dominant noise source. Use of object tracking allowsthe exposure time of a rolling shutter imager to be longer than wouldotherwise be possible. Such may advantageously allow use of ambientlight or lower levels of active lighting than might otherwise beemployed with a global shutter imager.

While some implementations may advantageously employ rolling shutterimagers, for example as the relatively narrow field-of-view imager,other implementations may employ a global shutter image for therelatively narrow field-of-view imager and/or for the relatively widefield-of-view imager. For example, a global shutter imager may beemployed as the relatively narrow field-of-view imager in the InCart™and ShelfHawk™ applications, discussed below.

The relatively smaller pixel size typical of image sensors of rollingshutter imagers may allow a smaller imager package size, with a shorterfocal length lens than might otherwise be required. Typically, for anequivalent f-number, the aperture diameter of a rolling shutter imageris smaller than that of a global shutter imager. This can advantageouslyreduce cost while increasing tracking speed.

The imaging system may be employed in loss prevention, for example inretail environments. For example, two imagers may be employed to coveran area (e.g., checkout station or counter, aisle, game pit), one imagerhaving a relatively wider field-of-view and the other having arelatively narrower field-of-view which is steerable. For instance therelatively narrower field-of-view may be steerable via a steeringmirror. The type of steering mirror, and in particular the speed atwhich the steering mirror can operate, may be a function of the specificapplication. For example, some applications may have relatively slowmoving objects to track or objects that move at a fixed speed (e.g., ona conveyor belt) or a known speed (e.g., speed sensed with one or moreencoders), with trigger signals received via one or more sensors ortransducers (e.g., photocells, inductive sensors, ultrasonic sensors).In such applications, the steering mirror may take the form of amechanical motorized platform that allows the imager (e.g., imagesensor) to move with respect to an optic assembly. For instance, devicesand techniques employed for stabilization in digital single lens reflect(DSLR) cameras may be suitable. Other applications may employ a faststeering mirror, for example the Eagle Eye™ fast steering mirror,developed by DataLogic.

This approach can be employed, for example, as a security camera inretail (e.g., grocery stores, convenience stores) or other locations,advantageously avoiding the need for additional security cameras. Thisapproach can be employed, for example, to advantageously detect objectsor items in a shopping cart, shopping basket, or elsewhere at a checkoutstation or counter. For instance, objects or items left in a shoppingcart or shopping basket may be intentionally or unintentionally removedfrom the retail environment generating an alert to retail personnel toprevent such from occurring. This approach can additionally oralternatively be employed, for example, to advantageously detect,prevent and otherwise deter what is called “sweet-hearting” by retailpersonnel such as checkout cashiers or clerks. In “sweet-hearting,” thecashier or clerk intentionally fails to scan or “ring up” an item, forexample pretending to scan or “ring the item up,” for customers who areknown by or colluding with the cashier or clerk. This allows thecustomer to steal items in a way that is typically very difficult todetect. Monitoring items at the checkout station or counter, forinstance on a conveyer belt, may prevent such. Further, this approachcan be employed, for example, to recognize individuals (e.g.,customers), including ones that may have previously been banned from thepremises, or to provide prompts to personnel allowing customers to begreeted by name or otherwise be recognized. This approach can beemployed, for example, to advantageously monitor shopping at locationsother than the checkout station or counter of a retail environment. Forexample, this approach can be employed, for example, to monitor or trackthe selection of items or objects from shelves and placement into ashopping cart or shopping basket in aisles of a retail environment,allowing better assessment of shopping patterns or purchasing decisionsor, conversely, detection of shop lifting. Such could likewise beemployed to monitor pick-and-place operations in a warehouseenvironment, or gaming wagering, card dealing or play in a gamingestablishment such as a casino.

An imaging system to image objects in an environment may be summarizedas including a first imager having a first imager field-of-view tocapture images of the environment; a steering mirror interposed along afirst optical path between the first imager and the environment, thesteering mirror selectively operable to steer at least a portion of thefirst imager field-of-view relative to one or more objects in theenvironment; and an object tracking subsystem that includes at least onehardware processor, the object tracking subsystem communicativelycoupled to cause the steering mirror to steer the first imagerfield-of-view based at least in part on information indicative of atleast one of a presence, a position, a speed or a direction of at leastone object in the environment to at least partially track objects in theenvironment. The first imager may be a rolling shutter imager. The firstimager may be a rolling shutter imager having a complementary metaloxide semiconductor (CMOS) image sensor and at least one of a mechanicalor electronic rolling shutter. The steering mirror may be a fastscanning mirror.

The object tracking subsystem may include a second imager having asecond imager field-of-view to capture images of the environment, thefirst imager field-of-view relatively more narrow than the second imagerfield-of-view; and a control subsystem, the control subsystemcommunicatively coupled the second imager to receive informationdirectly or indirectly therefrom, and communicatively coupled to causethe steering mirror to steer the first imager field-of-view based atleast in part on information received via the second imager. The objecttracking subsystem may detect an entrance of at least a first objectinto the second field-of-view. The object tracking subsystem mayidentify an object entering the second field-of-view as corresponding toa defined type of object. The object tracking subsystem may detect anentrance of at least a first object into the first field-of-view. Theobject tracking subsystem may identify an object entering the firstfield-of-view as corresponding to a defined type of object. The objecttracking subsystem may determine a position of the object. The objecttracking subsystem may determine at least an estimate of a speed of theobject. The object tracking subsystem may determine a direction of theobject. The object tracking subsystem may determine a direction of theobject.

The imaging system may further include a steering mirror actuatordrivingly coupled to the steering mirror and responsive to signals fromthe control subsystem of the object tracking subsystem to steer thefirst imager field-of-view.

The first imager may be a rolling shutter imager having an image sensorand at least one of a mechanical or electronic rolling shutter, thefirst imager having a higher resolution than the second imager, and mayfurther include a steering mirror actuator drivingly coupled to thesteering mirror and responsive to signals from the control subsystem ofthe object tracking subsystem to concurrently steer an entirefield-of-view of the image sensor of the first imager. The controlsubsystem of the object tracking system may cause the steering mirroractuator to move the steering mirror to an initial scan position at afirst speed, then to immediately follow a rolling shutter exposure bypanning the sensor at a second speed, the second speed slower than thefirst speed. The environment may be a retail environment. The retailenvironment may be a checkout stand with a point-of-sale terminal, andthe first and the second fields-of-view may encompass at least a portionof the checkout stand. The retail environment may be a checkout standwith a conveyor belt, and the first and the second fields-of-view mayencompass at least a portion of the conveyor belt. The retailenvironment may be an aisle spaced remotely from a checkout stand, andthe first and the second fields-of-view may encompass at least a portionof the aisle. The object tracking subsystem may identify a first objectin the first field-of-view as corresponding to a defined stock keepingunit (SKU) of a retail object. The object tracking subsystem mayidentify a second object in at least one of the first or the secondfields-of-view as corresponding to at least one of a shopping cart or ashopping basket in which one or more retail objects reside. The objecttracking subsystem may determine whether one or more retail objects areretained in a shopping cart or a shopping basket at a defined point in aretail transaction.

The imaging system may further include a variable focus lens in thefirst optical path between the first imager and the environment; anillumination source positioned and oriented to illuminate at least aportion of the environment in the first field-of-view; and a polarizerin the first optical path between the first imager and the environment.The system may compensate to recover blur and motion induced artifactsin rolling shutter imagers. The system may additionally or alternativelycoordinate operation of wide field-of-view FOV imagers and narrowfield-of-view imagers, for example generating coordinates of object ofinterest in the wide field-of-view that allows tracking in the narrowfield-of-view, tracking multiple objects concurrently, and selectingobjects including prioritizing objects.

A method of operation in an imaging system to image objects in anenvironment, the imaging system including a first imager having a firstimager field-of-view, a steering mirror interposed along a first opticalpath between the first imager and the environment, and an objecttracking subsystem that includes at least one hardware processor, may besummarized as including detecting at least one of a presence, aposition, a speed or a direction of at least one object in theenvironment by the object tracking subsystem; steering the first imagerfield-of-view relative to one or more objects in the environment basedat least in part on the detecting at least one of a presence, aposition, a speed or a direction of the at least one object in theenvironment to at least partially track objects in the environment; andcapturing images of the at least one object by the first imager as thefirst imager field-of-view is steered.

The method may further include operating the first imager as a rollingshutter imager. The imaging system may include a fast scanning mirror,and steering the first imager field-of-view may include moving the fastscanning mirror.

The object tracking subsystem may include a control subsystem thatincludes at least one hardware processor and a second imager, the secondimager having a second imager field-of-view of the environment, and themethod may further include receiving information by the controlsubsystem directly or indirectly from the second imager; and analyzingthe received information by the control subsystem to detect at least oneof the presence, the position, the speed or the direction of the atleast a first one of the objects in the environment. The detecting bythe object tracking subsystem may include detecting an entrance of atleast a first one of the at least one object into the secondfield-of-view. The detecting by the object tracking subsystem mayinclude identifying at least a first one of the at least one objectentering the second field-of-view as corresponding to a defined type ofobject. The detecting by the object tracking subsystem may includedetecting an entrance of at least a first one of the at least one objectinto the first field-of-view. The detecting by the object trackingsubsystem may include identifying at least a first one of the at leastone object entering the first field-of-view as corresponding to adefined type of object. The detecting by the object tracking subsystemmay include determining a position of the first one of the objects. Thedetecting by the object tracking subsystem may include determining atleast an estimate of a speed of the first one of the objects. The objecttracking subsystem may include determining a direction of the first oneof the objects. The detecting by the object tracking subsystem mayinclude determining a direction of the first one of the objects.

The method may further include providing a control signal by a controlsystem, directly or indirectly, to a steering mirror actuator which isdrivingly coupled to the steering mirror to steer the first imagerfield-of-view. Providing a control signal to a steering mirror actuatormay include providing control signals to the steering mirror actuator tomove to the steering mirror to an initial scan position at a firstspeed, then to follow a rolling shutter exposure by panning a sensor ata second speed, the second speed slower than the first speed. Theenvironment may be a retail environment and capturing images of the atleast one object by the first imager may include capturing images of alocation in the retail environment. Capturing images of a location inthe retail environment may include capturing images that encompass atleast a portion of a checkout stand in the retail environment. Capturingimages of a location in the retail environment may include capturingimages that encompass at least a portion of a conveyor belt in theretail environment. Capturing images of a location in the retailenvironment may include capturing images that encompass at least aportion of an aisle spaced remotely from a checkout stand in the retailenvironment.

The method may further include identifying a first object in the firstfield-of-view as corresponding to a defined stock keeping unit (SKU) ofa retail object.

The method may further include identifying a second object in at leastone of the first or the second fields-of-view as corresponding to atleast one of a shopping cart or a shopping basket in which one or moreretail objects reside.

The method may further include determining whether one or more retailobjects are retained in a shopping cart or a shopping basket at adefined point in a retail transaction.

The method may further compensate to recover blur and motion inducedartifacts in rolling shutter imagers. The method may additionally oralternatively coordinate operation of wide field-of-view FOV imagers andnarrow field-of-view imagers, for example generating coordinates of anobject of interest in the wide field-of-view that allows tracking in thenarrow field-of-view, tracking multiple objects concurrently, orselecting objects including prioritizing objects.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

In the drawings, identical reference numbers identify similar elementsor acts. The sizes and relative positions of elements in the drawingsare not necessarily drawn to scale. For example, the shapes of variouselements and angles are not necessarily drawn to scale, and some ofthese elements may be arbitrarily enlarged and positioned to improvedrawing legibility. Further, the particular shapes of the elements asdrawn, are not necessarily intended to convey any information regardingthe actual shape of the particular elements, and may have been solelyselected for ease of recognition in the drawings.

FIG. 1 is a schematic view of an imaging system, according to at leastone illustrated embodiment, which may be employed in tracking and/ormonitoring objects in an environment, for example a retail environment,the imaging system including a first imager and a second imager with awider field-of-view than that of the first imager, and an objecttracking subsystem to cause the field-of-view of the first imager totrack an object in the environment based on information from the secondimager.

FIG. 2 is an isometric view of an imager of an imaging system, such asthat of FIG. 1, which includes a rolling shutter imager comprising animage sensor and either a manual or electronic shutter, according to oneillustrated embodiment.

FIG. 3 is a schematic view of an imaging system, according to at leastone illustrated embodiment, which may be employed in tracking and/ormonitoring objects in an environment, for example a retail environment,the imaging system including a first imager and an object trackingsubsystem to cause the field-of-view of the first imager to track anobject in the environment based on information from one or morenon-image based sensors.

FIG. 4 is a top plan view of a portion of a retail environment in theform of a checkout station or counter, which is monitored via an imagingsystem such as that of FIG. 1 or 3, according to at least oneillustrated embodiment.

FIG. 5 is a top plan view of a portion of a retail environment in theform of an aisle with sets of shelves, which is monitored via an imagingsystem such as that of FIG. 1 or 3, according to at least oneillustrated embodiment.

FIG. 6 is a flow diagram of a high level method of operation in animaging system, according to at least one illustrated embodiment.

FIG. 7 is a flow diagram of a low level method of operation in animaging system, to capture images via a first imager while steering afirst imager field-of-view, according to at least one illustratedembodiment.

FIG. 8 is a flow diagram of a low level method of operation in animaging system to detect at least one of presence, position, speed ordirection of one or more objects in the environment, according to atleast one illustrated embodiment.

FIG. 9 is a flow diagram of a low level method of operation in animaging system to detect at least one of presence, position, speed ordirection of one or more objects in the environment, according to atleast one illustrated embodiment.

FIG. 10 is a flow diagram of a low level method of operation in animaging system to detect at least one of presence, position, speed ordirection of one or more objects in the environment, according to atleast one illustrated embodiment.

FIG. 11 is a flow diagram of a low level method of operation in animaging system to detect at least one of presence, position, speed ordirection of one or more objects in the environment, according to atleast one illustrated embodiment.

FIG. 12 is a flow diagram of a low level method of operation in animaging system to steer a first imager field-of-view 108 based at leastin part on detection to track one or more objects in the environment,according to at least one illustrated embodiment.

FIG. 13 is a flow diagram of a low level method of operation in animaging system to steer a first imager field-of-view based at least inpart on detection to track one or more objects in the environment,according to at least one illustrated embodiment.

DETAILED DESCRIPTION

In the following description, certain specific details are set forth inorder to provide a thorough understanding of various disclosedimplementations. However, one skilled in the relevant art will recognizethat implementations may be practiced without one or more of thesespecific details, or with other methods, components, materials, etc. Inother instances, well-known structures associated with imagers andimaging systems, cameras, computer systems, server computers, and/orcommunications networks have not been shown or described in detail toavoid unnecessarily obscuring descriptions of the implementations.

Unless the context requires otherwise, throughout the specification andclaims that follow, the word “comprising” is synonymous with“including,” and is inclusive or open-ended (i.e., does not excludeadditional, unrecited elements or method acts).

Reference throughout this specification to “one implementation” or “animplementation” means that a particular feature, structure orcharacteristic described in connection with the implementation isincluded in at least one implementation. Thus, the appearances of thephrases “in one implementation” or “in an implementation” in variousplaces throughout this specification are not necessarily all referringto the same implementation. Furthermore, the particular features,structures, or characteristics may be combined in any suitable manner inone or more implementations.

As used in this specification and the appended claims, the singularforms “a,” “an,” and “the” include plural referents unless the contentclearly dictates otherwise. It should also be noted that the term “or”is generally employed in its sense including “and/or” unless the contentclearly dictates otherwise.

The headings and Abstract of the Disclosure provided herein are forconvenience only and do not interpret the scope or meaning of theimplementations.

FIG. 1 shows an imaging system 100, according to at least oneillustrated embodiment. The imaging system 100 may be employed intracking and/or monitoring objects 102 (only one shown in FIG. 1) in anenvironment 104, for instance objects in a retail environment.

The imaging system 100 includes a first imager 106 having a first imagerfield-of-view 108 to capture images of the environment 104. The imagingsystem 100 includes a steering mirror 110 interposed along a firstoptical path, represented by line 112, between the first imager 106 andthe environment 104. The steering mirror 110 is selectively operable tosteer at least a portion of the first imager field-of-view 108 relativeto one or more objects 102 in the environment 104.

The imaging system 100 includes an object tracking subsystem 114 thatincludes one or more hardware processors. The object tracking subsystem114 is communicatively coupled to cause the steering mirror 110 to steerthe first imager field-of-view 108 based at least in part on informationindicative of at least one of an appearance, a presence, a position, aspeed or a direction of at least one object 102 in the environment 104to at least partially track objects 102 in the environment 104. Forexample, the object tracking subsystem 114 is communicatively coupled toprovide control signals to a steering mirror actuator 116 to cause thesteering mirror actuator 116 to move the steering mirror 110 to move thefirst field-of-view 108 of the first imager 106.

The steering mirror actuator 116 is drivingly coupled to the steeringmirror 110 and responsive to signals from a control subsystem 118 tosteer the first imager field-of-view 108. The steering mirror actuator116 may take any of a large variety of forms. For example, the steeringmirror actuator 116 may take the form of an electric motor, for instancea stepper motor. Also for example, the steering mirror actuator 116 maytake the form of a solenoid. Also for example, the steering mirroractuator 116 may take the form of one or more piezoelectric crystals orelements. Also for example, the steering mirror actuator 116 may takethe form of an electromagnetic and a magnetic element (e.g., magnet,ferrous metal). The fast steering mirror 110 may, for example, take theform of the Eagle Eye™ fast steering mirror, developed by DataLogic.

As best illustrated in FIG. 2, the first imager 106 preferably takes theform a rolling shutter imager 106. As compared to a “global shutter”imager, a rolling image shutter imager 106 advantageously employssmaller pixels and lower noise. The rolling shutter imager 106 mayinclude a complementary metal oxide semiconductor (CMOS) image sensor106 a and at least one of a mechanical or electronic rolling shutter 106b. The steering mirror 110 may pivot (as represented by double-headedarrow 120) about a pivot axis 122 to scan the field-of-view 208 of theimage sensor 106 a as indicated by double-headed arrows 124 a, 124 bbetween a first position 226 a and a second position 126 b. The steeringmirror 110 is a fast scanning mirror, for example the Eagle Eye™ faststeering mirror, developed by DataLogic. The steering mirror actuator116 may be responsive to signals from the control subsystem 118 toconcurrently steer an entire field-of-view 208 of the image sensor 106 aof the first imager 106. For example, the control subsystem 118 maycause the steering mirror actuator 116 to move the steering mirror 110to an initial scan position (e.g., 126 a) at a first speed, then toimmediately follow a rolling shutter exposure by panning thefield-of-view 208 of the first image sensor 106 a at a second speed, thesecond speed slower than the first speed.

Returning to FIG. 1, the first imager 106 may optionally include avariable focus lens 128 in the first optical path 112 between the firstimager 106 and the environment. 104. Additionally or alternatively, thefirst imager 106 may optionally include a polarizer 130 in the firstoptical path 112 between the first imager 106 and the environment 104.Additionally or alternatively, the first imager 106 or the imagingsystem 100 may optionally include an illumination source 132 positionedand oriented to illuminate at least a portion of the environment 104 inthe first imager field-of-view 108.

In the implementation illustrated in FIG. 1, the object trackingsubsystem 114 includes a second imager 134 having a second imagerfield-of-view 136 to capture images of the environment 104. Notably, thefirst imager field-of-view 108 is relatively narrower than the secondimager field-of-view 136.

The control subsystem 118 is communicatively coupled to the secondimager 134 to receive information directly or indirectly therefrom. Thecontrol subsystem 118 is communicatively coupled, e.g., via steeringmirror actuator 116, to cause the steering mirror 110 to steer the firstimager field-of-view 108 based at least in part on information receivedvia the second imager 134.

The control subsystem 118 may include one or more controllers orprocessors, for example one or more microcontrollers or microprocessors140, graphical processor units (GPUs) 142 a, 142 b, application specificintegrated circuits (ASICs), programmable logic units (PLUs) orprogrammable gate arrays (PGAs). The control subsystem 118 may includeone or more nontransitory storage media, for example one or morenon-volatile and/or volatile nontransitory storage media, for instanceone or more read only memories (ROM) 144, random access memories (RAM)146, registers, Flash memory 148, spinning magnetic media and drive,spinning optical media and drive, etc. The one or more nontransitorystorage media may store at least one of processor-executableinstructions and/or data, which when execute by one or more controllersor processors, causes the controller(s) or processor(s) to perform thealgorithms, methods and functions described herein.

The control subsystem 118 may further include one or more motorcontrollers 149 or other controllers communicatively coupled to controlone or more actuators, for instance steering mirror actuators 116. Thecontrol subsystem 118 may include one or more wired or wireless ports(not shown) to provide communications with various other elements ofcomponents of the imaging system 100, with other components in theenvironment 104 (e.g., POS terminal, backend inventory tracking system,SKU lookup system) or other components or systems outside theenvironment 104 (e.g., ordering system, customer tracking system orcustomer loyalty system). The control subsystem 118 may further includeone or more communications paths or channels, for example one or morebuses 150, for instance communications buses, power buses, command orinstruction buses, address buses, etc. The control subsystem 118 or aportion thereof may form a part of the object tracking subsystem 114.

In some implementations, the object tracking subsystem 114 detects anappearance or entrance of at least a first object 102 into the secondfield-of-view 136 of the second imager 134. Additionally oralternatively, the object tracking subsystem 114 identifies an object102 entering the second field-of-view 136 of the second imager 134 ascorresponding to a defined type of object (e.g., particular SKU).

Additionally or alternatively, in some implementations, the objecttracking subsystem 114 detects an appearance or entrance of at least afirst object 102 into the first field-of-view 108 of the first imager106. Additionally or alternatively, the object tracking subsystem 114identifies an object entering the first field-of-view 108 of the firstimager 106 as corresponding to a defined type of object (e.g.,particular SKU).

In some implementations, the object tracking subsystem 114 determines aposition of the object 102. The object tracking subsystem 114 determinesat least an estimate of a speed of the object 102. Additionally oralternatively, the object tracking subsystem 114 determines a directionof the object 102.

When employing a moving mirror in an object tracking subsystem, e.g.,the object tracking subsystem 114, there are several potential sourcesof motion jitter. For example, a moving mirror and/or imager or cameramay be mounted in a manner in which it is subjected to vibration withrespect to the object or item being imaged. For example, it is typicalto mount an imager or camera from a ceiling via a pole, which subjectsthe imager or camera to lateral vibrations. To address this type ofjitter, the system may employ a multi-axis accelerometer mounted todetect and measure motion of the imager or camera. A processor mayemploy the measure of motion to correct for the motion jitter in theimage, similar to motion or image correction techniques used insmartphones. Also for example, the steering or moving mirror itself maygive rise to a second type of motion jitter. A processor may usefeedback to eliminate or minimize this type of motion jitter to anydesired level. Any residual motion jitter may still be measurable in thesensor signal. In some instances the sensor signal itself may becorrupted with some noise, which limits the accuracy of the feedbacksystem, so cannot effectively be used as an indication or measure ofresidual motion jitter. As a further example, the motion of the objector item itself may give rise to motion jitter. For that type of motionjitter, the processor may track motion from frame to frame of imagescaptured by the wide field-of-view imager or camera. This may beimplemented via an optical flow algorithm, which can be atwo-dimensional or three-dimensional motion vector.

FIG. 3 shows an imaging system 300, according to at least oneillustrated embodiment. The imaging system 300 may be employed intracking and/or monitoring objects 302 on a conveyor belt 352 of aconveyor system 354 in an environment 104, for instance objects in aretail environment 104.

The conveyor system 354 includes a conveyor belt 352 that transports theobjects 125 in at least one direction (indicated by arrow 356), as oneor more drive wheels or cogs 358 a, 358 b rotate (illustrated bydouble-headed arrows 360). In some implementations, the conveyor belt352 may move in two, diametrically opposed, directions.

The imaging system 300 includes many components that are similar or evenidentical to those of imaging system 100 (FIG. 1). In the interest ofconciseness, only substantial differences are discussed below.

The imaging system 300 includes a first imager 106 with a first imagerfield-of-view 108. The first imager field-of-view 108 may extendlaterally across the path of the conveyor belt 352 at a first positionalong a path of the conveyor belt 352.

In contrast to the object tracking subsystem 114 of FIG. 1, the imagingsystem 300 includes an object tracking subsystem 314 that employssensors other than a second imager. For example, the object trackingsubsystem 314 may include one or more non-image based sensors 362 a, 362b that detect a presence or passing of one or more objects 102. Forinstance, one or more pairs of presence/absence sensors 362 a, 362 b(e.g., light emitter and light receiver pair) are positioned to detectpassage of an object 102, for instance on the conveyor belt 352 orsimilar device. As illustrated, each light emitter 364 a and lightreceiver 364 b (only one of each called out in FIG. 3) in a lightemitter and light receiver pair 362 a, 362 b are shown in tandem, nextto each other on a same side of the conveyor belt 352, with a respectivereflector or mirror 366 a, 366 b opposed across the conveyor belt 352from each light emitter and light receiver pair 362 a, 362 b to reflectlight emitted (arrow not called out) by the light emitter 364 a of thelight emitter and light receiver pair 362 a, 362 b back (arrow notcalled out) to the respective light receiver 364 b of the light emitterand light receiver pair 362 a, 362 b, creating a line of light acrossthe conveyor belt 352. Other configurations may be employed. Forexample, the light emitters 364 a and the light receivers 364 b may bepositioned one above the other. Alternatively, the light emitters 364 aand light receivers 364 b may be positioned across the width of theconveyor belt 352 from one another. In any of these implementations, theline of light is broken or interrupted by passage of an object 102,giving rise to a signal detectable by the respective light receiver 356b of the light emitter and light receiver pair 362 a, 362 b.

As illustrated, a first light emitter and light receiver pair 362 a maybe spaced at a first position 368 a along a path of travel (arrow 356)of the object 102 or conveyor belt 352, relatively upstream of the firstimager 106. A second light emitter and light receiver pair 362 b may bespaced at a second position 368 b along the path of travel (arrow 356)of the object 102 or conveyor belt 352, relatively upstream of the firstimager 106 and downstream of the first position 368 a, at a known fixeddistance d from the first position 368 a or first light emitter andlight receiver pair 362 a. The control subsystem 118 of the objectdetection system may receive signals from the first and the second lightemitter and receiver pairs 362 a, 362 b. The control subsystem 118 maydetermine an amount or period of time between the object passing thefirst light emitter and receiver pair 362 a or first position 368 a andthen passing the second light emitter and receiver pair 362 b or secondposition 368 b. The control subsystem 118 uses the timing informationand known distance d to determine any one or more of an appearance or apresence (e.g., detects passage at first or second positions), aposition (e.g., detects passage at first or second position and knowsspeed of conveyor belt or detects speed of object 102), a speed (e.g.,speed=distance/time) and/or a direction of travel (e.g., passes firstposition before passing second position, or passes second positionbefore passing first position) for the object 102. The control subsystem118 then generates signals to cause a first field-of-view 108 of thefirst imager 106 to at least partially track the object 102 over atleast a portion of travel of the object 102. This advantageously allowscapture of higher resolution pictures than might otherwise be possible,while reducing or eliminating blurring. This advantageously allowseffective use of lower cost, smaller imagers or image sensors than mightotherwise be possible.

FIG. 4 shows a first retail environment 104 a in which an imaging system100, 300 is employed, in the form of a checkout station or counter 470of a retail establishment, according to one illustrated embodiment.

The retail environment 104 a includes a first conveyor belt 452 on whicha customer 472 may unload objects 102 a-102 g (e.g., groceries, packagedgoods, retail items) (collectively 102), for example from a shoppingcart 474 and/or shopping basket 476. The first conveyor belt 452 movesin a direction indicated by arrow 456. A reader, for instance amachine-readable symbol reader (e.g., barcode symbol reader or scanner)478 may be positioned to read machine-readable symbols e.g., barcodesymbols) from the objects 102. The machine-readable symbol reader 478may, for example, be built into the first conveyor belt 452 or a counter470, as illustrated by the X-shaped window through which light (e.g.,laser beam(s)) is emitted and returned. Additionally or alternatively,the checkout station or counter 470 may include one or more hand-heldmachine-readable symbol readers (not illustrated).

The retail environment 104 includes a point-of-sale (POS) terminal 480,which may be operated by a person, such as a cashier or retailerrepresentative 482. The point-of-sale terminal 480 allows transactionsto be “rung up” or entered or otherwise executed, including acceptingcash payments. As is common in checkout stations or counters, one ormore customer facing terminals 484 may be provided to allow the customerto swipe credit or debit cards, execute Near Field Communications (NFC)financial transactions (e.g., ApplePar), enter personal identificationnumbers (PINs), enter or scan customer loyalty program information, etc.

The retail environment 104 may optionally include a loading area and/ora second conveyor belt 486 on which objects (e.g., groceries, packagedgoods, retail items) 102 may be placed, for example immediately beforebeing placed in bags. Typically, the cashier or retail representative482 places the objects 102 in the loading area or second conveyor belt486 after scanning or “ringing the objects 102 up” as part of thetransaction.

The retail environment 104 may include one or more first or primaryimagers (e.g., imagers with relatively high resolution and relativelynarrow fields-of-view) and one or more object tracking subsystems, forexample as illustrated and described above, for instance with respect toFIGS. 1 and 3 (not illustrated in FIG. 4 in the interest of drawingclarity). While non-imager based object tracking systems (e.g., objecttracking subsystem 314 of FIG. 3) may be employed, the embodiment ofFIG. 4 is discussed with reference to an imager based object trackingsystem 114, hence reference to a second imager 134 and a second imagerfield-of-view 436. Notably, the second imager field-of-view 436 is widerthan the first imager field-of-view.

Some implementations may include a respective object tracking subsystem114, 314 for each first or primary imager 106. Some implementations mayshare an object tracking subsystem 114, 314 between two or more first orprimary imagers 106. In such implementations, the object trackingsubsystem 114, 314 may include a second imager 134 that have afield-of-view 136, 436 that encompasses an area that includes areasencompassed by the respective fields-of-view 108, 408 of two or morefirst or primary imagers 106. Alternatively, in such implementations,the object tracking subsystem 114, 314 may include two or more secondimagers 134, that each have respective fields-of-view 136, 436 thatencompass respective ones of areas that are encompassed by therespective fields-of-view 108, 408 of two or more first or primaryimagers 106.

As illustrated in FIG. 4, the second imager has a second imagerfield-of-view 436 that encompasses a relatively larger area of thecheckout station or counter 470. When the object tracking subsystem 114detects and object of interest, e.g., object 102 b, the object trackingsubsystem 114 may provide information or instructions that steer thefirst field-of-view 408 of the first imager 106 (FIG. 1) to track theobject of interest, e.g., object 102 b. For example, the object trackingsystem 114 may identify objects 102, corresponding to a defined stockkeeping unit (SKU) of a retail objects, as objects of interest, e.g.,object 102 b. The objects 102 identified as being of interest may be inthe shopping cart 474, in the shopping basket 476, on the first conveyorbelt 452, in the loading area and/or on the second conveyor belt 486, oreven elsewhere in proximity of the checkout station or counter 470.

The imaging system 100, 300 (may use information discerned from thefirst imager 106 (FIG. 1) to, for example, determine whether one or moreretail objects 102 are retained in a shopping cart 474 or a shoppingbasket 476 at a defined point (e.g., at or past the machine-readablesymbol reader 478) in a retail transaction. The occurrence of this eventcan cause the imaging system 100, 300 (FIGS. 1, 3) to issue an alert ornotification and/or record information indicative of the event as apossible case of “inventory shrinkage,” shoplifting, or theft. Theimaging system 100, 300 (FIGS. 1, 3) may use information discerned fromthe first imager 106 (FIGS. 1, 3) to, for example, determine whether oneor more retail objects 102 are transferred by the cashier or retailrepresentative 482 to an area (e.g., loading area and/or on the secondconveyor belt 486) without being scanned or read, and hence withoutbeing tallied for a corresponding payment. This can cause the imagingsystem 100, 300 to issue an alert or notification and/or recordinformation indicative of the event as a possible case of “inventoryshrinkage,” shoplifting, or theft. The notifications or alerts may be avisual and/or aural notification or alert. The notifications or alertsmay be presented to a cashier or retail clerk and/or to a supervisor ormanager. The notifications or alerts may be provided via an electronicnotification, for instance electronic mail (email) or short messageservice (SMS) message or text.

FIG. 5 shows a retail environment 104 in which an imaging system 100,300 is employed, in the form of an aisle, remotely located from acheckout station or counter (FIG. 4) of a retail establishment,according to one illustrated embodiment.

The retail environment 104 includes a first set of shelves 588 a and asecond set of shelves 588 b opposed to the first set of shelves 588 a todefine an aisle 590 therebetween. The aisle 590 is typically spaced atleast a few feet or meters from a checkout station or counter 470 (FIG.4). The shelves 588 a, 588 b typically carry a large number of each of alarge variety of items or objects for sale (e.g., groceries, packagedgoods, retail items) (collectively 102).

A customer 472 may walk down the aisle 590 selecting desired objects 102for purchase. The customer 472 may, in some cases, employ a shoppingcart 474 or shopping basket 476 (illustrated in FIG. 4, not illustratedin FIG. 5) to assist in moving or transporting the items 102 h-102 k,eventually to the checkout station or counter 470 (FIG. 4).

As illustrated in FIG. 5, a second imager field-of-view 536 of thesecond imager 134 (FIG. 1, not illustrated in FIG. 5) encompasses arelatively large area of the aisle 590 and/or corresponding sets ofshelves 588 a, 588 b, for example covering a complete width W of theaisle 590 (e.g., dimension between opposed sets of shelves 588 a, 588 bor perpendicular to the set of shelves 588 a, 588 b). Less typically,the second imager field-of-view 536 covers a complete length L of theaisle 590 (e.g., dimension measured parallel to the opposed sets ofshelves 588 a, 588 b). When the object tracking subsystem 114 detects anobject of interest, e.g., object 1021, the object tracking subsystem 114may provide information or instructions that steer the firstfield-of-view 508 of the first imager 106 (FIGS. 1, 3) to track theobject of interest, e.g., object 1021. For example, the object trackingsystem 114 (FIG. 1) may identify objects 102 corresponding to a definedstock keeping unit (SKU) of retail objects as objects of interest, e.g.,object 1021. The objects 102 identified as being of interest may, forexample, be in the shopping cart 474, in the shopping basket 476 (FIG.4), held by the customer 472, or even on one of the shelves 588 a, 588b.

The imaging system 100, 300 may use information discerned from the firstimager 106 (FIGS. 1, 3) to, for example, determine whether one or moreretail objects 102 are retained in a shopping cart 474 or a shoppingbasket 476 (FIG. 4) or returned to a shelf 588 a, 588 b.

This can cause the imaging system 100, 300 to issue an alert ornotification and/or record information indicative of the event as apossible case of “inventory shrinkage,” shoplifting, or theft. Thenotification or alert may be a visual and/or aural notification oralert. The notification or alert may be presented to a cashier or retailclerk and/or to a supervisor or manager. The notification or alert maybe provided via an electronic notification, for instance electronic mail(email) or short message service (SMS) message or text.

This can cause the imaging system 100, 300 to track anonymizedpurchasing habits of customers 472, for instance the amount of timespent making a purchasing decision and/or the types of goods that arestudied and either purchased (e.g., placed or left in shopping cart 474or shopping basket 476) or not purchased (e.g., returned to shelves 588a, 588 b) after study by the customers 472. Such may also detectinstances or occurrence of shoplifting, for instance where an item isneither returned to a shelf 588 a, 588 b nor placed in a shopping cart474 or shopping basket 476.

FIG. 6 shows a method 600 of operation in an imaging system, forinstance the imaging system 100 of FIG. 1, according to at least oneillustrated embodiment.

The method 600 starts at 602. For example, the method 600 may start inresponse to a turning ON of the imaging system 100 or a componentthereof, or in response to invocation by a calling routine or program.

At 602, the imaging system 100, or a component thereof, detects at leastone of presence, position, speed or direction of one or more objects 102in an environment 104. For example, the imaging system 100 may includean object tracking subsystem 114 (FIG. 1) to track one or more objects102 in the environment 104. As discussed above, the object trackingsubsystem 114 may take a variety of forms, which employ one or moresensors to detect one or more characteristics of the object(s),including, for instance, appearance, position, location, speed, size,dimensions, images, and/or encoded information (e.g., optically readablemachine readable symbols, wirelessly readable (radio) wirelesstransponders such as radio frequency identification (RFID)transponders).

In one exemplary implementation, a second imager 134 (FIG. 1), whichincludes a second image sensor, is employed to capture images of asecond imager field-of-view 136 (FIG. 1) encompassing an area of theenvironment 104. The second imager field-of-view 136 encompasses arelatively larger area of the environment 104 than a first imagerfield-of-view 108 (FIG. 1) of a first imager 106 (FIG. 1).

In another exemplary implementation, one or more sensors detect theappearance, presence or passing of one or more objects. For instance,presence/absence sensors (e.g., light emitter and light receiver pair)362 a, 362 b (FIG. 3) may detect passage of an object 102, for instanceon a conveyor belt 352 or similar device. Two or more light emitter andlight receiver pairs 362 a, 362 b may be spaced along a path of travel356 (FIG. 3), and a period of time between passage by a first then asecond light emitter and light receiver pair 362 a, 362 b may indicatethe speed and directional of travel.

At 604, the imaging system 100, or a component thereof, steers a firstimager field-of-view 108, based at least in part on detection, to trackone or more objects in the environment 104.

At 606, a first imager 106 of the imaging system 100, or a componentthereof, captures images of the first imager field-of-view 108 as thefirst field-of-view 108 is steered.

At 608, the method 600 ends or terminates, for instance until invokedagain. Alternatively, the method 600 may periodically, aperiodicallyand/or continuously repeat, for instance until the imaging system 100 isturned OFF. The method 600 may be implemented in one or more threads,executing either concurrently in parallel or sequentially.

FIG. 7 shows a method 700 of operation in an imaging system, forinstance the imaging system 100 of FIG. 1, according to at least oneillustrated embodiment. The method 700 may be employed in execution ofthe method 600 (FIG. 6), for instance to capture images of the firstimager field-of-view 108 while steering the first imager field-of-view108 as at 606 of method 600 (FIG. 6).

At 702, an object tracking subsystem 114, 314 determines a position,orientation and/or rotational speed or timing for the fast scanningmirror in order to track one or more objects in an environment.

At 704, the object tracking subsystem 114, 314 generates or causes to begenerated, signals that are applied to move the fast scanning mirror totrack one or more objects in an environment. For example, the objecttracking subsystem 114, 314 may generate signals that drive a steeringmirror 110 via a motor controller 148 (FIG. 1) and a steering mirroractuator 116.

At 706, the control subsystem, or another component of the imagingsystem 100, 300, operates a rolling shutter of the first imager 106 tosequentially capture images of the environment that is within afield-of-view 108, 308 of the first imager 106.

FIG. 8 shows a method 800 of operation in an imaging system, forinstance the imaging system 100 of FIG. 1, according to at least oneillustrated embodiment. The method 800 may be employed in execution ofthe method 600 (FIG. 6), for instance to detect at least one ofpresence, position, speed or direction of one or more object in theenvironment 104 as at 602 of method 600 (FIG. 6).

At 802, a second imager 134 (FIG. 1) captures images of a second imagerfield-of-view 136. The second imager field-of-view 136 is typicallywider or encompasses a larger area than the first imager field-of-view108.

At 804, the control subsystem 118 receives information from one or moreobject tracking sensor, for example, the control subsystem 118 mayreceive information from a second imager 134 (FIG. 1) having a widerfield-of-view 136 than a first field-of-view 108 of the first imager106.

At 806, the control subsystem 118 analyzes the received information todetect an appearance, presence, position, speed and/or direction oftravel of one or more objects 102 in the environment 104, for exampleanalyzing information received from the second imager 134 to detect atleast one of appearance, presence, position, speed and/or direction oftravel of one or more objects 102.

At 808, the control subsystem 118 provides control signals to cause asteering mirror actuator 116 to steer the first imager field-of-view 108based on the detected appearance, presence, position, speed and/ordirection of travel of one or more objects 102.

FIG. 9 shows a method 900 of operation in an imaging system, forinstance the imaging system 100 of FIG. 1, according to at least oneillustrated embodiment. The method 900 may be employed in execution ofthe method 600 (FIG. 6), for instance to detect at least one ofappearance, presence, position, speed or direction of one or moreobjects 102 in the environment 104 as at 602 of method 600 (FIG. 6).

At 902, an object tracking system 114, 314, or a component thereof,detects an appearance or entrance of one or more objects into the secondimager field-of-view.

At 904, the object tracking system 114 (FIG. 1), or a component thereof,identifies one or more objects entering the second field-of-view 136 ascorresponding to a defined type of object. Such may employ any one ormore of a variety of techniques. For example, the object tracking system114 may employ three-dimensional images and/or structured light todetermine a set of values that define the shape and size of theobject(s). Also for example, the object tracking system 114 may employcolor image processing to identify unique graphical elements, textand/or symbols on the objects. The object tracking system 114 may usevarious machine-vision techniques (e.g., Sobel filter) on image data,and/or may use various parcel dimensioning techniques to identifyobjects as corresponding to a particular class of objects (e.g.,restricted sale items such as a case of beer, carton of cigarettes). Theobject tracking system 114 may query a database that storesrelationships between characteristic object shapes and dimensions andobject descriptors, identifiers or other information specific to thetype of object (e.g., price, manufacturer, model, SKU, age restrictionson sale of the product, special promotions, discounts, tax status).Additionally or alternatively, the object tracking system 114 may querya database that stores relationships between characteristic objectpackaging color(s), logos, text, symbols or graphics and objectdescriptors, identifiers or other information specific to the type ofobject (e.g., price, manufacturer, model, SKU, age restrictions on saleof the product, special promotions, discounts, tax status).

FIG. 10 shows a method 1000 of operation in an imaging system, forinstance the imaging system 100 of FIG. 1, according to at least oneillustrated embodiment. The method 1000 may be employed in execution ofthe method 600 (FIG. 6), for instance to detect at least one ofappearance, presence, position, speed or direction of one or more object102 in the environment 104 as at 602 of the method 600 (FIG. 6). Incontrast to the method 900 illustrated in FIG. 9, the method 1000employs the first imager to detect at least one of appearance, presence,position, speed or direction of one or more objects 102 in anenvironment.

At 1002, an object tracking system 114, 314 (FIGS. 1, 3), or a componentthereof, detects an appearance or entrance of one or more objects 102into a first field-of-view 108 of a first imager 106.

At 1004, the object tracking system 114, or a component thereof,identifies one or more objects 102 entering the first imagerfield-of-view 108 of the first imager 106 as corresponding to one ormore defined types of objects 102. Such may employ any one or more of avariety of techniques. For example, the object tracking system 114 mayemploy three-dimensional images and/or structured light to determine aset of values that define the shape and size of the object(s). Also forexample, the object tracking system 114 may employ color imageprocessing to identify unique graphical elements, text and/or symbols onthe objects. The object tracking system 114 may use variousmachine-vision techniques (e.g., Sobel filter) on image data, and/or mayuse various parcel dimensioning techniques to identify objects ascorresponding to a particular class of objects (e.g., case of beer,carton of cigarettes). The object tracking system 114 may query adatabase that stores relationships between characteristic object shapesand dimensions and object descriptors, identifiers or other informationspecific to the type of object (e.g., price, manufacturer, model, SKU,age restrictions on sale of the product, special promotions, discounts,tax status). Additionally or alternatively, the object tracking system114 may query a database that stores relationships betweencharacteristic object packaging color(s), logos, text, symbols orgraphics and object descriptors, identifiers or other informationspecific to the type of object (e.g., price, manufacturer, model, SKU,age restrictions on sale of the product, special promotions, discounts,tax status).

FIG. 11 shows a method 1100 of operation in an imaging system, forinstance the imaging system 100 of FIG. 1, according to at least oneillustrated embodiment. The method 1100 may be employed in execution ofthe method 600 (FIG. 6), for instance to detect at least one ofpresence, position, speed or direction of one or more objects 102 in theenvironment 104 as at 602 at method 600 (FIG. 6).

At 1102, an object tracking system 114, 314 (FIGS. 1, 3), or a componentthereof, determines a position of one or more objects 102 in anenvironment 104.

At 1104, the object tracking system 114, 314, or a component thereof,determines at least an estimate of speed of the object(s) 102. Theobject tracking system 114 may, for example, use successive images takenat a known time apart to determine a change in position of the object102. Knowing the distance traveled by the object in a known period oftime allows the object tracking system 114 to determine a speed of anobject 102 that is in motion. The information also allows the objecttracking system 114 to determine any of a location, position, and/ordirection of travel of the object 102 that is in motion. Alternatively,the object tracking system 314 may, for example, evaluate an amount ofblurriness in an image, and estimate speed and/or direction of travel ofan object 102 based at least in part on the evaluation of blurriness.Alternatively, the object tracking system 314 may, for example, detectpassage of the object 102 past a first position and a second positionwhich are a known distance apart. The object tracking system 314 maydetermine a time between the successive passages by the two positions.Knowing the distance traveled by the object in a known period of timeallows the object tracking system 314 to determine a speed of an object102 that is in motion. The information also allows the object trackingsystem 314 to determine any of a location, position, and/or direction oftravel of the object 102 that is in motion.

At 1106, the object tracking system 114, 314, or a component thereof,determines the direction of travel of one or more objects(s) 102. Forexample, the object tracking system 114 may use successive images todetermine a change in position of the object 102, and hence a directionof travel of the object 102. Also for example, the object trackingsystem 114 may, for example, detect passage of the object 102 past afirst position and a second position which have a known relativeposition (e.g., upstream, downstream) to determine a direction of travelby detecting which position is passed in which order.

FIG. 12 shows a method of operation in an imaging system, for instancethe imaging system 100 of FIG. 1, according to at least one illustratedembodiment. The method 1200 may be employed in execution of the method600 (FIG. 6), for instance to steer a first imager field-of-view 108based at least in part on detection to track one or more objects in theenvironment 104 as at 602 of method 600 (FIG. 6).

At 1202, an object tracking system 114, 314 moves a steering mirror 110to an initial scan position, at a first speed. For example, the objecttracking system 114, 314 may supply signals to control a tracking mirroractuator 116 (FIG. 1) to move quickly into a first position inpreparation for tracking an object 102, for example an object 102 thatis in motion, or to orient toward an object 102 that is not in motion.

At 1204, an object tracking system 114, 314 moves the steering mirror110, at second speed, to follow a rolling shutter exposure. The secondspeed is typically slower than the first speed.

FIG. 13 shows a method 1300 of operation in an imaging system, forinstance the imaging system 100 of FIG. 1, according to at least oneillustrated embodiment. The method 1200 may be employed in execution ofthe method 600 (FIG. 6), for instance to steer a first imagerfield-of-view 108 based at least in part on detection to track one ormore objects in the environment 104 as at 602 of method 600 (FIG. 6).

At 1302, the first imager (FIGS. 1, 3) and/or the second imager 136(FIG. 1) capture images of at least portion of conveyor belt 352, 452(FIGS. 3, 4) of checkout stand 470 (FIG. 4) in retail environment 104 a.

At 1304, the first imager 106 and/or the second imager 136 captureimages of at least portion of aisle 590 in retail environment 104 b.

At 1306, the object tracking subsystem 114 identifies a first object 102in the first field-of-view 108, 408, 508 (FIGS. 1, 3, 4 and 5) ascorresponding to a defined stock keeping unit (SKU) of a retail object.The object tracking subsystem 114 may employ any of a large variety ofmachine-vision or image processing techniques, for example determining ashape, and/or dimensions of the object 102, shapes and/or dimensions ofportions of the object 102, text, graphics, symbols on the object 102,reading machine-readable symbols, interrogating wireless transponders,for instance RFID transponders, etc. The object tracking system 114 mayuse various machine-vision techniques (e.g., Sobel filter) on imagedata, and/or may use various parcel dimensioning techniques to identifyobjects as corresponding to a particular class of objects (e.g., case ofbeer, carton of cigarettes). The object tracking system 114 may query adatabase that stores relationships between characteristic object shapesand dimensions and object descriptors, identifiers or other informationspecific to the type of object (e.g., price, manufacturer, model, SKU,age restrictions on sale of the product, special promotions, discounts,tax status). Additionally or alternatively, the object tracking system114 may query a database that stores relationships betweencharacteristic object packaging color(s), logos, text, symbols orgraphics and object descriptors, identifiers or other informationspecific to the type of object (e.g., price, manufacturer, model, SKU,age restrictions on sale of the product, special promotions, discounts,tax status).

At 1308, the object tracking subsystem 114, or some other component, maycompare an object or item identified from the image(s) to a recently(e.g., most-recently) scanned or “rung up” object 102 or item anddetermine whether they match at 1310. Thus, for example, if an object oritem 102 appearing just past the machine-readable symbol reader 478(FIG. 4) or on the loading area and/or on the second conveyor belt 486or some other defined location, does not match a scanned or “rung up”object or item 102, the imaging system 100, 300 (FIGS. 1, 3) may issue anotification or alert at 1312 indicating that the object or item 102 wasnot yet scanned or “rung up.” The notification or alert may be a visualand/or aural notification or alert. The notification or alert may bepresented to a cashier or retail clerk and/or to a supervisor ormanager. The notification or alert may be provided via an electronicnotification, for instance electronic mail (email) or short messageservice (SMS) message or text.

At 1314, the object tracking subsystem 114 identifies a second object102 in at least one of the first field-of-view 108, 408, 508 (FIGS. 1,3, 4, 5) or the second field-of-view 136, 436, 536 (FIGS. 1, 4 and 5) ascorresponding to at least one of a shopping cart 474 or a shoppingbasket 476 in which one or more retail objects 102 reside. The objecttracking subsystem 114 may use any of the previously describedtechniques to identify the second object (e.g., shopping cart 474,shopping basket 476).

At 1316, the object tracking subsystem 114 compares the first and thesecond objects to determine whether one or more retail objects 102 areretained in a shopping cart 474 or a shopping basket 476 at a definedpoint in a retail transaction. For example, the object trackingsubsystem 114 may determine whether a boundary of a first object 102resides in a boundary of the second object 474, 476 in one or moretwo-dimensional or three-dimensional images. Alternatively, the objecttracking subsystem 114 may determine whether a centroid of a firstobject 102 resides in a boundary of the second object 474, 476 in one ormore two-dimensional or three-dimensional images. At 1318, the imagingsystem 100, 300 (FIGS. 1, 3) determines whether a retail object 102remains in a shopping cart 474 or shopping basket. In response to adetermination that a retail object 102 remains in a shopping cart 474 orshopping basket 476, the imaging system 100, 300 or some other componentmay issue a notification or alert at 1320 indicating that the object oritem 102 remains in the shopping cart or shopping basket and was not yetscanned or “rung up.” The notification or alert may be a visual and/oraural notification or alert. The notification or alert may be presentedto a cashier or retail clerk and/or to a supervisor or manager. Thenotification or alert may be provided via an electronic notification,for instance electronic mail (email) or short message service (SMS)message or text.

Example 1 InCart™ System Application

The InCart™ system is a security system that looks at items in shoppingcarts and/or shopping baskets and/or elsewhere at a checkout area orcheckout lane, for instance while a customer is checking out of a retailenvironment (e.g., supermarket, grocery store, convenience store, bigbox store or other retail or warehouse environment). The InCart™ systemmay employ a wide field-of-view and a narrow field-of-view to trackitems in the checkout area or checkout lane including those in shoppingcarts or shopping baskets, on conveyors or on or at machine-readablesymbol readers (e.g., barcode symbol scanners). For example, one or moreimagers or cameras are mounted, typically in the ceiling, looking downinto the shopping cart and/or at the checkout lane. As the shopping cartmoves forward in the checkout lane, a narrow field-of-view is moved totrack the shopping cart and/or items. For the narrow field-of-view anEagle Eye tracking image capture system may be used, with a narrowfield-of-view imager or camera and tracking mirror. Image processingtechniques may be employed to detect items in the shopping cart, if any.Image processing techniques may be employed to detect items on or at themachine-readable symbol reader (e.g., barcode symbol scanner) and/or onthe checkout belt to ensure all items are scanned properly.

The InCart™ system may employ at least one wide field-of-view camera,with a field-of-view or fields-of-view that encompass a whole “frontend” (e.g., checkout area or checkout lane) including shopping carts,shopping baskets, conveyor belts, loading areas, customers, cashier. Oneor more processor-based devices (e.g., programmed computer with one ormore microprocessors and/or graphic process units) executeprocessor-executable instructions and/or data stored in or on one ormore non-transitory processor-readable media to perform imageprocessing. The processor-based device can determine regions-of-interest(ROI), for example the shopping cart, items on check out conveyor beltor in specific areas such as loading areas upstream or downstream of amachine-readable symbol reader or scanner. The imager or camera istypically mounted in or proximate the ceiling, relatively far away(e.g., 14-22 feet) from the items to be imaged. The items to be imagedin the shopping cart and on the conveyor belt are all about 3 feet abovethe ground, and thus can effectively be considered planar. Consequently,the system may be considered or operated as a two-dimensional imagesystem, even if using a three-dimensional imager or camera (e.g.,time-of-flight camera) or even if using structured lighting.

The processor-based device may, for example, employ scale-Invariantfeature transform (Sift) algorithms or packages, allowing theprocessor-based device to easily match the images or view of the narrowfield-of-view (“zoomed in” Eagle Eye™ tracking camera system) with theimages or view of the wide field-of-view imager or camera. If the centerof the image from the narrow field-of-view imager or camera is at angiven X, Y position in the image from the wide field-of-view camera, andneeds to move 100 pixels in a given direction (e.g., up), then thiswould be a known distance/offset, and the system can command thetracking mirror (e.g., tracking mirror of Eagle Eye™ tracking imagingsystem) to move accordingly. A map of coordinates may be built up andstored, that provides a relationship between each X, Y pair ofcoordinates in the wide field-of-view images or reference frame and thecorresponding x, y pair of coordinates in the narrow field-of-viewimages or reference frame. Tracking in two-dimensions provides asimplified scenario, although a similar approach may be employed forthree-dimensions.

In operation, the system may capture narrow field-of-view images ofevery item on the checkout belt, either before or after the respectiveitem is scanned or rung up (e.g., machine-readable symbol read). Thesystem can additionally analyze images of each shopping cart or shoppingbasket, identifying the contents (i.e., each item) thereof. In someimplementations, there may be glare from the overhead lights overheadwhen the imager or camera is positioned overhead. Tracking the items toget a good image may advantageously overcome the problem presented byglare. In some implementations, the system may capture, for exampleapproximately 2 to 3 narrow field-of-view images of each item in thecheckout area or checkout lane (e.g., in shopping cart, in shoppingbasket, on conveyor, at scanner, in load area) per transaction. Themapping between coordinate systems of the wide field-of-view imager orcamera and the narrow field-of-view imager or camera may need to be moreaccurate to capture images of items in motion than to capture images ofstatic items.

Example 2 ShelfHawk™ Application

In the ShelfHawk™ application, imagers or cameras are positioned andoriented to image one or more shelves, for example shelves in an aisleof a supermarket, grocery store, convenience store, big box store orother retail or warehouse environment. The ShelfHawk™ system may includeone or more processor-based systems that execute at least one ofprocessor-readable instructions or data stored in one or morenontransitory processor readable media. The processor-based system mayanalyze images of the shelf or shelves, determining when one or moreitems normally stocked on the shelf or shelves is out of stock, and inresponse providing a notification or alert to store personnel and/or toan automated ordering system. In this application, it may be sufficientif the narrow field-of-view (e.g., Eagle Eye™ imager or camera) is movedor scanned across the shelf or shelves in a defined pattern, for examplescanning the shelf or shelves, or portion thereof, from left to right,and/or from top to bottom. The processor-based system analyzes the widefield-of-view images captured by a wide field-of-view imager or cameramainly to detect an absence of items or a change of state, causing thenarrow field-of-view imager or camera to capture “zoomed” images fromthe appropriate positions on the shelf or shelves.

As explained above, since the shelf or shelves are essentially planar,the image processing can effectively be treated as a two-dimensionaltransformation. To build a mapping between the coordinate systems (e.g.,X, Y Cartesian coordinates) of the wide field-of-view images and thenarrow field-of-view images the processor-based system may cause thetracking mirror of the Eagle Eye™ system to move the narrowfield-of-view in a defined pattern, for example from left to right, fromtop to bottom (i.e., raster scan pattern), over the wide imagefield-of-view. The processor-based system can thus build a map of X, Ypositions in the narrow field-of-view coordinates. Alternatively, theprocessor-based system may employ a simple linear mapping.

In the ShelfHawk™ application, the wide field-of-view imager or cameracan be operated to autonomously repeatedly grab images (e.g., one imageper second, or one image per 10 seconds). The processor-based device mayperform image processing to determine and if there are any changesdetected on the shelf between wide field-of-view images. In response toa detected change, the processor-based device may cause thefield-of-view of the narrow field-of-view imager or camera (e.g., EagleEye™ system) to move to an appropriate position or orientation andcapture a narrow field-of-view (“zoomed in”) image of that portion ofthe shelf or shelves. If the image captured by the narrow field-of-viewimager or camera is a bit off of position as compared to the widefield-of-view imager, the processor-based device may adjust the positionor orientation of the tracking mirror, and hence the position of thenarrow field-of-view accordingly. The processor-based device may onceagain compare the image captured by the wide field-of-view imager orcamera to the image captured by the narrow field-of-view imager orcamera, for example using a SiFT algorithm or package. The ShelfHawk™application tends to be time insensitive.

Coordinating the wide field-of-view with targeting the narrowfield-of-view (e.g., Eagle Eye™) via a tracking or moving mirror orother technique could be implemented in a variety of ways. For example,one approach is to employ “up-to-date” images of the whole areaencompassed by the field-of-view of the wide field-of-view imager orcamera. Images of static objects or items within the wide field-of-viewonly need to be captured infrequently. However, images of moving objectsor items within the wide field-of-view need to be captured (i.e.,updated) more rapidly than those of the static objects. This can beimplemented by a processor-based device that identifiesregions-of-interest (ROI) that correspond to objects or items in motion,and which executes a background task that slowly pans through thefield-of-view to handle the static objects, and a foreground task thatcaptures images of the regions-of-interest (ROI). The processor-baseddevice may employ various parameters and thresholds to identify theregions-of-interest, for example motion, direction of motion, speed(e.g., motion above a first threshold rate), etc. Further, processing inthe foreground task may be prioritized based on various criteria, forinstance prioritized in order of: speed of motion and/or prioritized bysize of object or item.

Additionally or alternatively, for many applications there is a sense ofwhat is important, which can be advantageously used by theprocessor-based system. For example, for a system that performs overheadpeople tracking which looks at what people are picking up, it isimportant to identify and focus on a person's hands. This may beimplemented in a fairly straightforward manner, for instance performingdepth segmentation on three-dimensional image information captured viaan overhead three-dimensional imager or camera. The processor-baseddevice may Identify a person's hand or hands in the wide field-of-viewimagers, and use the information to cause the field-of-view of thenarrow field-of-view camera to move to capture images of the regionaround people's hands, for example via a tracking mirror. Theprocessor-based system can track multiple objects, for example via afast steering mirror bouncing between locations from frame-to-frame.When using a relatively slower steering mirror, it may be advantageousfor the processor-based system to perform path planning to reduce thetotal path time needed to capture images of all identified objects. Forexample, the processor-based system may determine or generate a paththat moves from place-to-place in a circular path instead of bouncingback and forth between objects in a star pattern.

The foregoing detailed description has set forth various implementationsof the devices and/or processes via the use of block diagrams,schematics, and examples. Insofar as such block diagrams, schematics,and examples contain one or more functions and/or operations, it will beunderstood by those skilled in the art that each function and/oroperation within such block diagrams, flowcharts, or examples can beimplemented, individually and/or collectively, by a wide range ofhardware, software, firmware, or virtually any combination thereof. Inone implementation, the present subject matter may be implemented viaApplication Specific Integrated Circuits (ASICs). However, those skilledin the art will recognize that the implementations disclosed herein, inwhole or in part, can be equivalently implemented in standard integratedcircuits, as one or more computer programs running on one or morecomputers (e.g., as one or more programs running on one or more computersystems), as one or more programs running on one or more controllers(e.g., microcontrollers) as one or more programs running on one or moreprocessors (e.g., microprocessors), as firmware, or as virtually anycombination thereof, and that designing the circuitry and/or writing thecode for the software and or firmware would be well within the skill ofone of ordinary skill in the art in light of this disclosure.

Those of skill in the art will recognize that many of the methods oralgorithms set out herein may employ additional acts, may omit someacts, and/or may execute acts in a different order than specified.

In addition, those skilled in the art will appreciate that themechanisms taught herein are capable of being distributed as a programproduct in a variety of forms, and that an illustrative implementationapplies equally regardless of the particular type of signal bearingmedia used to actually carry out the distribution. Examples of signalbearing media include, but are not limited to, the following: recordabletype media such as floppy disks, hard disk drives, CD ROMs, digitaltape, and computer memory.

The various implementations described above can be combined to providefurther implementations. To the extent that they are not inconsistentwith the specific teachings and definitions herein, all of the U.S.patents, U.S. patent application publications, U.S. patent applications,foreign patents, foreign patent applications and non-patent publicationsreferred to in this specification, including U.S. provisional patentapplication Ser. No. 62/222,595, filed Sep. 23, 2015, are incorporatedherein by reference, in their entirety. Aspects of the implementationscan be modified, if necessary, to employ systems, circuits and conceptsof the various patents, applications and publications to provide yetfurther implementations.

These and other changes can be made to the implementations in light ofthe above-detailed description. In general, in the following claims, theterms used should not be construed to limit the claims to the specificimplementations disclosed in the specification and the claims, butshould be construed to include all possible implementations along withthe full scope of equivalents to which such claims are entitled.Accordingly, the claims are not limited by the disclosure.

1. An imaging system to image objects in an environment, the imagingsystem comprising: a first imager having a first imager field-of-view tocapture images of the environment; a steering component selectivelyoperable to steer at least a portion of the first imager field-of-viewrelative to one or more objects in the environment; and an objecttracking subsystem that includes at least one hardware processor, theobject tracking subsystem communicatively coupled to cause the steeringcomponent to steer the first imager field-of-view based at least in parton information indicative of at least one of a presence, a position, aspeed or a direction of at least one object in the environment to atleast partially track objects in the environment.
 2. The imaging systemof claim 1 wherein the first imager is a rolling shutter imager.
 3. Theimaging system of claim 1 wherein the first imager is a rolling shutterimager having a complementary metal oxide semiconductor (CMOS) imagesensor and at least one of a mechanical or electronic rolling shutter.4. The imaging system of claim 1 wherein steering component comprises anelectro-actuated component that moves the image sensor to track theobject movement.
 5. The imaging system of claim 2 wherein the steeringcomponent comprises a steering mirror interposed along a first opticalpath between the first imager and the environment, the steering mirrorselectively operable to steer at least a portion of the first imagerfield-of-view relative to one or more objects in the environment, andwherein the steering mirror is a fast scanning mirror.
 6. The imagingsystem of claim 5 wherein the object tracking subsystem comprises: asecond imager having a second imager field-of-view to capture images ofthe environment, the first imager field-of-view relatively more narrowthan the second imager field-of-view; and a control subsystem, thecontrol subsystem communicatively coupled the second imager to receiveinformation directly or indirectly therefrom, and communicativelycoupled to cause the steering mirror to steer the first imagerfield-of-view based at least in part on information received via thesecond imager.
 7. The imaging system of claim 6 wherein the objecttracking subsystem detects an entrance of at least a first object intothe second field-of-view.
 8. The imaging system of claim 6 wherein theobject tracking subsystem identifies an object entering the secondfield-of-view as corresponding to a defined type of object.
 9. Theimaging system of claim 6 wherein the object tracking subsystem detectsan entrance of at least a first object into the first field-of-view. 10.The imaging system of claim 6 wherein the object tracking subsystemidentifies an object entering the first field-of-view as correspondingto a defined type of object.
 11. The imaging system of claim 7 whereinthe object tracking subsystem determines a position of the object. 12.The imaging system of any of claim 7 wherein the object trackingsubsystem determines at least an estimate of a speed of the object. 13.The imaging system of claim 12 wherein the object tracking subsystemdetermines a direction of the object.
 14. The imaging system of any ofclaim 7 wherein the object tracking subsystem determines a direction ofthe object.
 15. The imaging system of claim 7, further comprising: asteering mirror actuator drivingly coupled to the steering mirror andresponsive to signals from the control subsystem of the object trackingsubsystem to steer the first imager field-of-view.
 16. The imagingsystem of any of claim 7 wherein the first imager is a rolling shutterimager having an image sensor and at least one of a mechanical orelectronic rolling shutter, the first imager having a higher resolutionthan the second imager, and further comprising: a steering mirroractuator drivingly coupled to the steering mirror and responsive tosignals from the control subsystem of the object tracking subsystem toconcurrently steer an entire field-of-view of the image sensor of thefirst imager.
 17. The imaging system of claim 16 wherein the controlsubsystem of the object tracking system causes the steering mirroractuator to move the steering mirror to an initial scan position at afirst speed, then to immediately follow a rolling shutter exposure bypanning the sensor at a second speed, the second speed slower than thefirst speed.
 18. The imaging system of claim 6 wherein the environmentis a retail environment.
 19. The imaging system of claim 18 wherein theretail environment is a checkout stand with a point-of-sale terminal,and the first and the second fields-of-view each encompass at least aportion of the checkout stand.
 20. The imaging system of claim 18wherein the retail environment is a checkout stand with a conveyor belt,and the first and the second fields-of-view encompass at least a portionof the conveyor belt.
 21. The imaging system of claim 18 wherein theretail environment is an aisle spaced remotely from a checkout stand,and the first and the second fields-of-view encompass at least a portionof the aisle.
 22. The imaging system of any of claim 19 wherein theobject tracking subsystem identifies a first object in the firstfield-of-view as corresponding to a defined stock keeping unit (SKU) ofa retail object.
 23. The imaging system of any of claim 19 wherein theobject tracking subsystem identifies a second object in at least one ofthe first or the second field-of-view as corresponding to at least oneof a shopping cart or a shopping basket in which one or more retailobjects reside.
 24. The imaging system of any of claim 19 wherein theobject tracking subsystem determines whether one or more retail objectsare retained in a shopping cart or a shopping basket at a defined pointin a retail transaction.
 25. The imaging system of claim 6 wherein theenvironment is an industrial environment with a conveyor belt and thefirst field of view encompasses at least a portion of the conveyor belt.26. The imaging system of claim 1, further comprising: a variable focuslens positioned between the first imager and the environment; anillumination source positioned and oriented to illuminate at least aportion of the environment in the first field-of-view; and a polarizerpositioned between the first imager and the environment.
 27. The imagingsystem of claim 1 wherein the at least one processor performscompensation to recover blur and motion induced artifacts.
 28. Theimaging system of any of claim 5 wherein the at least one processoridentifies at least one region-of-interest in an image captured by thesecond imager, and determines a set of coordinates to steer the firstimager field-of-view to capture images of the region-of-interest. 29.The imaging system of claim 28 wherein the at least one processorgenerates a path to steer the first imager field-of-view to captureimages of multiple regions-of-interest.
 30. The imaging system of claim29 wherein the at least one processor prioritizes image capture by thefirst imager between multiple regions-of-interest.
 31. The imagingsystem of any of claim 5 wherein the second imager is a time-of-flightcamera. 32.-54. (canceled)